Advanced Course in Machine Learning

The problem of characterizing learnability is the most basic question of learning theory. A fundamental and long-standing answer, formally proven for supervised classification and regression, is that learnability is equivalent to uniform convergence, and that if a problem is learnable, it is learnable via empirical risk minimization. Furthermore, for the problem of binary classification, uniform convergence is equivalent to finite VC dimension. In this lecture we will talk about other methods for obtaining generalization bounds and establishing learnability. We start with PAC-Bayes bounds which can be though of as an extension to Minimum Description Length (MDL) bounds and Occam’s razor. Next, we discuss a compression bound which states that if a learning algorithm only uses a small fraction of the training set to form its hypothesis then it generalizes. Finally, we turn to online-to-batch conversions. In the next lecture we will discuss the “General Learning Setting” (introduced by Vapnik), which includes most statistical learning problems as special cases.